Evaluation of p-curve simulation results
In the following plots, always the RMSE goodness-of-fit measure is used (not chi2).
Some synthetic p-curves: Testing specific scenarios
Test a pure bump
Multiple DVs
Caution
I thought, the multiDV cannot produce a bump? Is this still MC error, or is there a real bump?
The best fitting scenarios from the “multiple DVs” simulations are:
Code
simres |> filter(type == "multDV") |> arrange(rmseBump) |> slice(1:5) |> select(c(1:9, 10:14), rmseBump) |> round(2) |> print() nvar r d prop_Hacker prop_H1 het iter alpha strategy p1 p2 p3 p4
1 150 0.9 0.2 0.75 0.1 0.4 10000 0.05 3 0.14 0.11 0.15 0.23
2 70 0.9 0.4 1.00 0.1 0.2 10000 0.05 3 0.15 0.11 0.15 0.23
3 150 0.9 0.2 1.00 0.5 0.0 10000 0.05 3 0.15 0.10 0.16 0.23
4 150 0.9 0.2 1.00 0.2 0.4 10000 0.05 3 0.18 0.10 0.15 0.22
5 150 0.9 0.3 0.75 0.2 0.2 10000 0.05 3 0.20 0.11 0.14 0.21
p5 rmseBump
1 0.37 0.04
2 0.36 0.04
3 0.36 0.05
4 0.35 0.05
5 0.34 0.05Optional Stopping
Code
# all optStop scenarios:
plot_pcurves(simres |> filter(type == "optStop"), poriginal = reference, n_best = 10) + ggtitle("Optional Stopping")
The best fitting scenarios from the “optional stopping” simulations are:
Code
simres |> filter(type == "optStop") |> arrange(rmseBump) |> slice(1:5) |> select(c(18:20, 10:14), rmseBump) |> round(2) |> print() nmin nmax stepsize p1 p2 p3 p4 p5 rmseBump
1 444 1063 1 0.16 0.15 0.15 0.15 0.39 0.01
2 50 444 1 0.15 0.13 0.16 0.16 0.40 0.01
3 177 1063 1 0.15 0.14 0.14 0.14 0.43 0.01
4 177 1063 5 0.15 0.14 0.14 0.18 0.39 0.02
5 177 444 1 0.17 0.15 0.16 0.14 0.37 0.02Test a “true signal + bump” pattern
Multiple DVs
Code
reference2 <- c(.40, .20, .10, .05, .25)
#reference <- c(.45, .22, .14, .05, .14)
# all multDV scenarios:
simres$rmseBump2 <- rmse(reference2, simres[, paste0("p", 1:5)])
plot_pcurves(simres |> filter(type == "multDV"), poriginal = reference2, n_best = 10, GOF="rmse")
The best fitting scenarios from the “multiple DVs” simulations are:
Code
simres |> filter(type == "multDV") |> arrange(rmseBump2) |> slice(1:5) |> select(c(1:9, 10:14), rmseBump2) |> round(2) |> print() nvar r d prop_Hacker prop_H1 het iter alpha strategy p1 p2 p3 p4
1 10 0.9 0.4 0.10 0.1 0.0 10000 0.05 1 0.38 0.20 0.15 0.12
2 10 0.9 0.4 0.10 0.1 0.0 10000 0.05 2 0.38 0.20 0.15 0.12
3 2 0.0 0.4 0.25 0.1 0.2 10000 0.05 1 0.41 0.18 0.13 0.13
4 2 0.0 0.4 0.25 0.1 0.2 10000 0.05 2 0.41 0.18 0.13 0.13
5 5 0.6 0.5 0.50 0.1 0.2 10000 0.05 1 0.40 0.18 0.14 0.12
p5 rmseBump2
1 0.16 0.06
2 0.16 0.06
3 0.16 0.06
4 0.16 0.06
5 0.15 0.06Optional Stopping
Code
# all optStop scenarios:
plot_pcurves(simres |> filter(type == "optStop"), poriginal = reference2, n_best = 10, GOF="rmse")
The best fitting scenarios from the “optional stopping” simulations are:
Code
simres |> filter(type == "optStop") |> arrange(rmseBump2) |> slice(1:5) |> select(c(18:20, 10:14), rmseBump2) |> round(2) |> print() nmin nmax stepsize p1 p2 p3 p4 p5 rmseBump2
1 444 1063 1 0.43 0.14 0.10 0.08 0.26 0.03
2 177 444 1 0.40 0.14 0.13 0.09 0.24 0.04
3 444 1063 1 0.38 0.15 0.11 0.11 0.25 0.04
4 177 1063 1 0.40 0.14 0.11 0.10 0.25 0.04
5 177 444 1 0.41 0.15 0.11 0.11 0.22 0.04Three empirical p-curves
“Sotola”
Perfect set (no p-hacking, real effects)
Code
plot_pcurves(simres |> filter(condition == "multDV_perfect"), poriginal = pcurves[pcurves$dataset == "sotola", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola multDV perfect")
Code
simres |> filter(condition == "multDV_perfect") |> arrange(rmseSotola) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseSotola) |> round(3) |> print() nvar r d prop_Hacker prop_H1 het iter alpha strategy nmin nmax stepsize
1 1 0 0.8 0 0.2 0.2 10000 0.05 1 NA NA NA
2 1 0 0.8 0 0.2 0.2 10000 0.05 2 NA NA NA
3 1 0 0.8 0 0.2 0.2 10000 0.05 3 NA NA NA
4 1 0 0.3 0 0.7 0.4 10000 0.05 1 NA NA NA
5 1 0 0.3 0 0.7 0.4 10000 0.05 2 NA NA NA
p1 p2 p3 p4 p5 rmseSotola
1 0.666 0.105 0.092 0.064 0.074 0.024
2 0.666 0.105 0.092 0.064 0.074 0.024
3 0.666 0.105 0.092 0.064 0.074 0.024
4 0.651 0.120 0.088 0.066 0.075 0.028
5 0.651 0.120 0.088 0.066 0.075 0.028H0 set (no p-hacking)
Code
plot_pcurves(simres |> filter(condition == "multDV_H0"), poriginal = pcurves[pcurves$dataset == "sotola", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola multDV H0")Warning: Removed 35 rows containing missing values or values outside the scale range
(`geom_line()`).
Code
simres |> filter(condition == "multDV_H0") |> arrange(rmseSotola) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseSotola) |> round(3) |> print() nvar r d prop_Hacker prop_H1 het iter alpha strategy nmin nmax stepsize
1 1 0 0 0 0 0 10000 0.05 1 NA NA NA
2 1 0 0 0 0 0 10000 0.05 2 NA NA NA
3 1 0 0 0 0 0 10000 0.05 3 NA NA NA
p1 p2 p3 p4 p5 rmseSotola
1 0.211 0.209 0.183 0.217 0.179 0.226
2 0.211 0.209 0.183 0.217 0.179 0.226
3 0.211 0.209 0.183 0.217 0.179 0.226multDV realistic set
Code
plot_pcurves(simres |> filter(condition == "multDV_realistic"), poriginal = pcurves[pcurves$dataset == "sotola", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola multDV realistic")
Code
simres |> filter(condition == "multDV_realistic") |> arrange(rmseSotola) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseSotola) |> round(3) |> print() nvar r d prop_Hacker prop_H1 het iter alpha strategy nmin nmax stepsize
1 5 0.9 0.8 0.50 0.2 0.2 10000 0.05 1 NA NA NA
2 5 0.9 0.8 0.50 0.2 0.2 10000 0.05 2 NA NA NA
3 5 0.3 0.6 0.10 0.3 0.2 10000 0.05 1 NA NA NA
4 5 0.3 0.6 0.10 0.3 0.2 10000 0.05 2 NA NA NA
5 5 0.9 0.8 0.75 0.5 0.0 10000 0.05 3 NA NA NA
p1 p2 p3 p4 p5 rmseSotola
1 0.643 0.107 0.099 0.066 0.084 0.022
2 0.643 0.107 0.099 0.066 0.084 0.022
3 0.635 0.113 0.107 0.065 0.081 0.023
4 0.635 0.113 0.107 0.065 0.081 0.023
5 0.662 0.102 0.091 0.069 0.076 0.025multDV worst set
Code
plot_pcurves(simres |> filter(condition == "multDV_worst"), poriginal = pcurves[pcurves$dataset == "sotola", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola multDV worst")
Code
simres |> filter(condition == "multDV_worst") |> arrange(rmseSotola) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseSotola) |> round(3) |> print() nvar r d prop_Hacker prop_H1 het iter alpha strategy nmin nmax stepsize
1 150 0 0 0.10 0 0 10000 0.05 1 NA NA NA
2 150 0 0 0.10 0 0 10000 0.05 2 NA NA NA
3 150 0 0 0.25 0 0 10000 0.05 1 NA NA NA
4 150 0 0 0.25 0 0 10000 0.05 2 NA NA NA
5 150 0 0 0.50 0 0 10000 0.05 1 NA NA NA
p1 p2 p3 p4 p5 rmseSotola
1 0.596 0.182 0.091 0.076 0.055 0.059
2 0.596 0.182 0.091 0.076 0.055 0.059
3 0.701 0.176 0.056 0.038 0.028 0.062
4 0.701 0.176 0.056 0.038 0.028 0.062
5 0.747 0.179 0.045 0.019 0.010 0.078optStop realistic set
Code
plot_pcurves(simres |> filter(condition == "optStop_realistic"), poriginal = pcurves[pcurves$dataset == "sotola", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola optStop realistic")
Code
simres |> filter(condition == "optStop_realistic") |> arrange(rmseSotola) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseSotola) |> round(3) |> print() nvar r d prop_Hacker prop_H1 het iter alpha strategy nmin nmax stepsize
1 NA NA 0.6 0.10 0.3 0.2 NA NA NA 177 1063 50
2 NA NA 0.7 0.25 0.2 0.2 NA NA NA 444 1063 100
3 NA NA 0.8 0.10 0.2 0.0 NA NA NA 50 1063 50
4 NA NA 0.5 0.25 0.3 0.4 NA NA NA 444 1063 100
5 NA NA 0.5 0.25 0.3 0.4 NA NA NA 177 444 50
p1 p2 p3 p4 p5 rmseSotola
1 0.660 0.099 0.099 0.067 0.075 0.021
2 0.663 0.092 0.089 0.069 0.088 0.023
3 0.643 0.100 0.098 0.076 0.084 0.024
4 0.670 0.091 0.085 0.067 0.088 0.025
5 0.656 0.103 0.092 0.072 0.077 0.025optStop worst set
Code
plot_pcurves(simres |> filter(condition == "optStop_worst"), poriginal = pcurves[pcurves$dataset == "sotola", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola optStop worst")
Code
simres |> filter(condition == "optStop_worst") |> arrange(rmseSotola) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseSotola) |> round(3) |> print() nvar r d prop_Hacker prop_H1 het iter alpha strategy nmin nmax stepsize
1 NA NA 0 0.1 0 0 NA NA NA 50 177 100
2 NA NA 0 0.1 0 0 NA NA NA 22 444 100
3 NA NA 0 0.1 0 0 NA NA NA 177 444 50
4 NA NA 0 0.1 0 0 NA NA NA 177 444 100
5 NA NA 0 0.1 0 0 NA NA NA 50 444 100
p1 p2 p3 p4 p5 rmseSotola
1 0.227 0.204 0.162 0.204 0.204 0.219
2 0.209 0.179 0.206 0.175 0.232 0.225
3 0.208 0.172 0.222 0.215 0.182 0.227
4 0.206 0.206 0.186 0.189 0.212 0.227
5 0.210 0.205 0.184 0.220 0.181 0.227“Wetzels”
Perfect set (no p-hacking, real effects)
Code
plot_pcurves(simres |> filter(condition == "multDV_perfect"), poriginal = pcurves[pcurves$dataset == "wetzels", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola multDV perfect")
Code
simres |> filter(condition == "multDV_perfect") |> arrange(rmseWetzels) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseWetzels) |> round(3) |> print() nvar r d prop_Hacker prop_H1 het iter alpha strategy nmin nmax stepsize
1 1 0 0.5 0 0.5 0.4 10000 0.05 1 NA NA NA
2 1 0 0.5 0 0.5 0.4 10000 0.05 2 NA NA NA
3 1 0 0.5 0 0.5 0.4 10000 0.05 3 NA NA NA
4 1 0 0.7 0 0.3 0.2 10000 0.05 1 NA NA NA
5 1 0 0.7 0 0.3 0.2 10000 0.05 2 NA NA NA
p1 p2 p3 p4 p5 rmseWetzels
1 0.688 0.107 0.074 0.076 0.055 0.004
2 0.688 0.107 0.074 0.076 0.055 0.004
3 0.688 0.107 0.074 0.076 0.055 0.004
4 0.686 0.112 0.081 0.065 0.057 0.006
5 0.686 0.112 0.081 0.065 0.057 0.006H0 set (no p-hacking)
Code
plot_pcurves(simres |> filter(condition == "multDV_H0"), poriginal = pcurves[pcurves$dataset == "wetzels", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola multDV H0")Warning: Removed 35 rows containing missing values or values outside the scale range
(`geom_line()`).
Code
simres |> filter(condition == "multDV_H0") |> arrange(rmseWetzels) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseWetzels) |> round(3) |> print() nvar r d prop_Hacker prop_H1 het iter alpha strategy nmin nmax stepsize
1 1 0 0 0 0 0 10000 0.05 1 NA NA NA
2 1 0 0 0 0 0 10000 0.05 2 NA NA NA
3 1 0 0 0 0 0 10000 0.05 3 NA NA NA
p1 p2 p3 p4 p5 rmseWetzels
1 0.211 0.209 0.183 0.217 0.179 0.238
2 0.211 0.209 0.183 0.217 0.179 0.238
3 0.211 0.209 0.183 0.217 0.179 0.238multDV realistic set
Code
plot_pcurves(simres |> filter(condition == "multDV_realistic"), poriginal = pcurves[pcurves$dataset == "wetzels", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola multDV realistic")
Code
simres |> filter(condition == "multDV_realistic") |> arrange(rmseWetzels) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseWetzels) |> round(3) |> print() nvar r d prop_Hacker prop_H1 het iter alpha strategy nmin nmax stepsize
1 2 0.0 0.7 0.50 0.5 0.0 10000 0.05 3 NA NA NA
2 2 0.9 0.6 0.25 0.5 0.2 10000 0.05 1 NA NA NA
3 2 0.9 0.6 0.25 0.5 0.2 10000 0.05 2 NA NA NA
4 10 0.0 0.6 0.75 0.5 0.2 10000 0.05 1 NA NA NA
5 10 0.0 0.6 0.75 0.5 0.2 10000 0.05 2 NA NA NA
p1 p2 p3 p4 p5 rmseWetzels
1 0.690 0.112 0.074 0.072 0.052 0.002
2 0.684 0.117 0.074 0.072 0.054 0.002
3 0.684 0.117 0.074 0.072 0.054 0.002
4 0.688 0.116 0.076 0.070 0.049 0.002
5 0.688 0.116 0.076 0.070 0.049 0.002multDV worst set
Code
plot_pcurves(simres |> filter(condition == "multDV_worst"), poriginal = pcurves[pcurves$dataset == "wetzels", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola multDV worst")
Code
simres |> filter(condition == "multDV_worst") |> arrange(rmseWetzels) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseWetzels) |> round(3) |> print() nvar r d prop_Hacker prop_H1 het iter alpha strategy nmin nmax stepsize
1 150 0 0 0.25 0 0 10000 0.05 1 NA NA NA
2 150 0 0 0.25 0 0 10000 0.05 2 NA NA NA
3 150 0 0 0.10 0 0 10000 0.05 1 NA NA NA
4 150 0 0 0.10 0 0 10000 0.05 2 NA NA NA
5 150 0 0 0.50 0 0 10000 0.05 1 NA NA NA
p1 p2 p3 p4 p5 rmseWetzels
1 0.701 0.176 0.056 0.038 0.028 0.035
2 0.701 0.176 0.056 0.038 0.028 0.035
3 0.596 0.182 0.091 0.076 0.055 0.051
4 0.596 0.182 0.091 0.076 0.055 0.051
5 0.747 0.179 0.045 0.019 0.010 0.052optStop realistic set
Code
plot_pcurves(simres |> filter(condition == "optStop_realistic"), poriginal = pcurves[pcurves$dataset == "wetzels", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola optStop realistic")
Code
simres |> filter(condition == "optStop_realistic") |> arrange(rmseWetzels) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseWetzels) |> round(3) |> print() nvar r d prop_Hacker prop_H1 het iter alpha strategy nmin nmax stepsize
1 NA NA 0.6 0.75 0.7 0.4 NA NA NA 22 177 100
2 NA NA 0.5 0.50 0.5 0.0 NA NA NA 8 177 100
3 NA NA 0.6 0.50 0.5 0.4 NA NA NA 50 177 50
4 NA NA 0.6 0.25 0.7 0.4 NA NA NA 22 1063 100
5 NA NA 0.8 0.10 0.3 0.4 NA NA NA 22 177 50
p1 p2 p3 p4 p5 rmseWetzels
1 0.687 0.115 0.075 0.069 0.054 0.003
2 0.682 0.115 0.076 0.072 0.055 0.003
3 0.687 0.113 0.070 0.073 0.057 0.004
4 0.687 0.114 0.076 0.068 0.055 0.004
5 0.687 0.109 0.071 0.076 0.056 0.004optStop worst set
Code
plot_pcurves(simres |> filter(condition == "optStop_worst"), poriginal = pcurves[pcurves$dataset == "wetzels", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola optStop worst")
Code
simres |> filter(condition == "optStop_worst") |> arrange(rmseWetzels) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseWetzels) |> round(3) |> print() nvar r d prop_Hacker prop_H1 het iter alpha strategy nmin nmax stepsize
1 NA NA 0 0.10 0 0 NA NA NA 50 177 100
2 NA NA 0 0.10 0 0 NA NA NA 50 444 100
3 NA NA 0 0.25 0 0 NA NA NA 22 177 100
4 NA NA 0 0.10 0 0 NA NA NA 177 444 100
5 NA NA 0 0.10 0 0 NA NA NA 177 444 50
p1 p2 p3 p4 p5 rmseWetzels
1 0.227 0.204 0.162 0.204 0.204 0.231
2 0.210 0.205 0.184 0.220 0.181 0.239
3 0.207 0.200 0.195 0.207 0.191 0.240
4 0.206 0.206 0.186 0.189 0.212 0.241
5 0.208 0.172 0.222 0.215 0.182 0.241“Simonsohn”
Perfect set (no p-hacking, real effects)
Code
plot_pcurves(simres |> filter(condition == "multDV_perfect"), poriginal = pcurves[pcurves$dataset == "simonsohn", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola multDV perfect")
Code
simres |> filter(condition == "multDV_perfect") |> arrange(rmseSimonsohn) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseSimonsohn) |> round(3) |> print() nvar r d prop_Hacker prop_H1 het iter alpha strategy nmin nmax stepsize
1 1 0 0.3 0 0.3 0.2 10000 0.05 1 NA NA NA
2 1 0 0.3 0 0.3 0.2 10000 0.05 2 NA NA NA
3 1 0 0.3 0 0.3 0.2 10000 0.05 3 NA NA NA
4 1 0 0.2 0 0.7 0.0 10000 0.05 1 NA NA NA
5 1 0 0.2 0 0.7 0.0 10000 0.05 2 NA NA NA
p1 p2 p3 p4 p5 rmseSimonsohn
1 0.474 0.169 0.137 0.102 0.118 0.038
2 0.474 0.169 0.137 0.102 0.118 0.038
3 0.474 0.169 0.137 0.102 0.118 0.038
4 0.427 0.188 0.145 0.122 0.118 0.041
5 0.427 0.188 0.145 0.122 0.118 0.041H0 set (no p-hacking)
Code
plot_pcurves(simres |> filter(condition == "multDV_H0"), poriginal = pcurves[pcurves$dataset == "simonsohn", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola multDV H0")Warning: Removed 35 rows containing missing values or values outside the scale range
(`geom_line()`).
Code
simres |> filter(condition == "multDV_H0") |> arrange(rmseSimonsohn) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseSimonsohn) |> round(3) |> print() nvar r d prop_Hacker prop_H1 het iter alpha strategy nmin nmax stepsize
1 1 0 0 0 0 0 10000 0.05 1 NA NA NA
2 1 0 0 0 0 0 10000 0.05 2 NA NA NA
3 1 0 0 0 0 0 10000 0.05 3 NA NA NA
p1 p2 p3 p4 p5 rmseSimonsohn
1 0.211 0.209 0.183 0.217 0.179 0.136
2 0.211 0.209 0.183 0.217 0.179 0.136
3 0.211 0.209 0.183 0.217 0.179 0.136multDV realistic set
Code
plot_pcurves(simres |> filter(condition == "multDV_realistic"), poriginal = pcurves[pcurves$dataset == "simonsohn", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola multDV realistic")
Code
simres |> filter(condition == "multDV_realistic") |> arrange(rmseSimonsohn) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseSimonsohn) |> round(3) |> print() nvar r d prop_Hacker prop_H1 het iter alpha strategy nmin nmax stepsize
1 70 0.3 0.2 0.10 0.3 0.2 10000 0.05 1 NA NA NA
2 70 0.3 0.2 0.10 0.3 0.2 10000 0.05 2 NA NA NA
3 70 0.0 0.1 0.10 0.1 0.2 10000 0.05 1 NA NA NA
4 70 0.0 0.1 0.10 0.1 0.2 10000 0.05 2 NA NA NA
5 70 0.3 0.1 0.25 0.7 0.0 10000 0.05 1 NA NA NA
p1 p2 p3 p4 p5 rmseSimonsohn
1 0.451 0.207 0.136 0.103 0.103 0.031
2 0.451 0.207 0.136 0.103 0.103 0.031
3 0.462 0.226 0.142 0.088 0.082 0.031
4 0.462 0.226 0.142 0.088 0.082 0.031
5 0.475 0.209 0.149 0.083 0.085 0.031multDV worst set
Code
plot_pcurves(simres |> filter(condition == "multDV_worst"), poriginal = pcurves[pcurves$dataset == "simonsohn", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola multDV worst")
Code
simres |> filter(condition == "multDV_worst") |> arrange(rmseSimonsohn) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseSimonsohn) |> round(3) |> print() nvar r d prop_Hacker prop_H1 het iter alpha strategy nmin nmax stepsize
1 150 0.3 0 0.1 0 0 10000 0.05 1 NA NA NA
2 150 0.3 0 0.1 0 0 10000 0.05 2 NA NA NA
3 70 0.0 0 0.1 0 0 10000 0.05 1 NA NA NA
4 70 0.0 0 0.1 0 0 10000 0.05 2 NA NA NA
5 70 0.3 0 1.0 0 0 10000 0.05 1 NA NA NA
p1 p2 p3 p4 p5 rmseSimonsohn
1 0.459 0.207 0.145 0.107 0.082 0.038
2 0.459 0.207 0.145 0.107 0.082 0.038
3 0.412 0.263 0.143 0.102 0.080 0.044
4 0.412 0.263 0.143 0.102 0.080 0.044
5 0.431 0.235 0.155 0.110 0.069 0.044optStop realistic set
Code
plot_pcurves(simres |> filter(condition == "optStop_realistic"), poriginal = pcurves[pcurves$dataset == "simonsohn", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola optStop realistic")
Code
simres |> filter(condition == "optStop_realistic") |> arrange(rmseSimonsohn) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseSimonsohn) |> round(3) |> print() nvar r d prop_Hacker prop_H1 het iter alpha strategy nmin nmax stepsize
1 NA NA 0.4 0.25 0.2 0.0 NA NA NA 8 177 100
2 NA NA 0.4 0.10 0.5 0.0 NA NA NA 8 177 50
3 NA NA 0.2 0.10 0.7 0.2 NA NA NA 50 444 100
4 NA NA 0.4 0.50 0.5 0.0 NA NA NA 8 177 50
5 NA NA 0.3 0.25 0.7 0.0 NA NA NA 50 177 100
p1 p2 p3 p4 p5 rmseSimonsohn
1 0.461 0.184 0.133 0.103 0.119 0.033
2 0.474 0.188 0.135 0.098 0.106 0.034
3 0.474 0.180 0.140 0.097 0.110 0.035
4 0.466 0.191 0.134 0.105 0.104 0.035
5 0.437 0.193 0.149 0.107 0.113 0.035optStop worst set
Code
plot_pcurves(simres |> filter(condition == "optStop_worst"), poriginal = pcurves[pcurves$dataset == "simonsohn", paste0("p", 1:5)], n_best=10) + ggtitle("Sotola optStop worst")
Code
simres |> filter(condition == "optStop_worst") |> arrange(rmseSimonsohn) |> slice(1:5) |> select(c(1:9, 18:20, 10:14), rmseSimonsohn) |> round(3) |> print() nvar r d prop_Hacker prop_H1 het iter alpha strategy nmin nmax stepsize
1 NA NA 0 0.10 0 0 NA NA NA 50 177 100
2 NA NA 0 0.10 0 0 NA NA NA 177 444 100
3 NA NA 0 0.10 0 0 NA NA NA 22 444 100
4 NA NA 0 0.50 0 0 NA NA NA 8 177 100
5 NA NA 0 0.25 0 0 NA NA NA 22 177 100
p1 p2 p3 p4 p5 rmseSimonsohn
1 0.227 0.204 0.162 0.204 0.204 0.128
2 0.206 0.206 0.186 0.189 0.212 0.135
3 0.209 0.179 0.206 0.175 0.232 0.137
4 0.200 0.212 0.185 0.192 0.212 0.137
5 0.207 0.200 0.195 0.207 0.191 0.138